{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Data Science\n",
    "## Text classification\n",
    "Author: Robert Moakler\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import the libraries we will be using\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn import metrics\n",
    "from sklearn import cross_validation\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "import matplotlib.pylab as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = 10, 8\n",
    "\n",
    "np.random.seed(36)\n",
    "\n",
    "# We will want to keep track of some different roc curves, lets do that here\n",
    "tprs = []\n",
    "fprs = []\n",
    "roc_labels = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "We have a new data set in `data/spam_ham.csv`. Let's take a look at what it contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!head -2 data/spam_ham.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we have two features: some text (looks like an email), and a label for spam or ham. What is the distribution of the target variable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cut -f2 -d',' data/spam_ham.csv | sort | uniq -c | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't look like that did what we wanted. Can you see why?\n",
    "\n",
    "The data in this file has **text data**. The text data in the first column can have commas. The command line will have some issues reading this data since it will try to split on all instances of the delimeter. Ideally, we would like to have a way of **encapsulating** the first column. Note that we actually have something like this in the data. The first column is wrapped in single quotes. Python (and pandas) have more explicit ways of dealing with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"data/spam_ham.csv\", quotechar=\"'\", escapechar=\"\\\\\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, we specify that fields that need to be encapsulated are done so with single quotes (`quotechar`). But, what if the text in this field uses single quotes? For example, apostrophes in words like \"can't\" would break the encapsulation. To overcome this, we **escape** single quotes that are actually just text. Here, we specify the escape character as a backslash (`escapechar`). So now, for example, \"can't\" would be written as \"can\\'t\".\n",
    "\n",
    "Let's take another look at our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the target is whether or not a record should be considered as spam. This is recorded as the string 'spam' or 'ham'. To make it a little easier for our classifier, let's recode it as `0` or `1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data['spam'] = pd.Series(data['spam'] == 'spam', dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are going to do some modeling, we should split our data into a training and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = data['text']\n",
    "Y = data['spam']\n",
    "\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, train_size=.75)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text as features\n",
    "How can we turn the large amount of text for each record into useful features?\n",
    "\n",
    "\n",
    "#### Binary representation\n",
    "One way is to create a matrix that uses each word as a feature and keeps track of whether or not a word appears in a document/record. You can do this in sklearn with a `CountVectorizer()` and setting `binary` to `true`. The process is very similar to how you fit a model: you will fit a `CounterVectorizer()`. This will figure out what words exist in your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "binary_vectorizer = CountVectorizer(binary=True)\n",
    "binary_vectorizer.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the vocabulary the `CountVectorizer()` learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "binary_vectorizer.vocabulary_.keys()[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know what words are in the data, we can transform our blobs of text into a clean matrix. Simply `.transform()` the raw data using our fitted `CountVectorizer()`. You will do this for the training and test data. What do you think happens if there are new words in the test data that were not seen in the training data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train_binary = binary_vectorizer.transform(X_train)\n",
    "X_test_binary = binary_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at our new `X_test_counts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_test_binary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sparse matrix? Where is our data?\n",
    "\n",
    "If you look at the output above, you will see that it is being stored in a *sparse* matrix (as opposed to the typical dense matrix) that is ~2k rows long and ~70k columns. The rows here are records in the original data and the columns are words. Given the shape, this means there are ~140m cells that should have values. However, from the above, we can see that only ~220k cells (~0.15%) of the cells have values! Why is this?\n",
    "\n",
    "To save space, sklearn uses a sparse matrix. This means that only values that are not zero are stored! This saves a ton of space! This also means that visualizing the data is a little trickier. Let's look at a very small chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_test_binary[0:20, 0:20].todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Applying a model\n",
    "Now that we have a ton of features (since we have a ton of words!) let's try using a logistic regression model to predict spam/ham."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train_binary, Y_train)\n",
    "\n",
    "print \"Area under the ROC curve on the test data = %.3f\" % metrics.roc_auc_score(model.predict(X_test_binary), Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this any good? What do we care about in this case? Let's take a look at our ROC measure in more detail by looking at the actual ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = metrics.roc_curve(Y_test, model.predict_proba(X_test_binary)[:,1])\n",
    "tprs.append(tpr)\n",
    "fprs.append(fpr)\n",
    "roc_labels.append(\"Default Binary\")\n",
    "ax = plt.subplot()\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlabel(\"fpr\")\n",
    "plt.ylabel(\"tpr\")\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlim([0, 1])\n",
    "plt.ylim([0, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Counts instead of binary\n",
    "Instead of using a 0 or 1 to represent the occurence of a word, we can use the actual counts. We do this the same way as before, but now we leave `binary` set to `false` (the default value)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fit a counter\n",
    "count_vectorizer = CountVectorizer()\n",
    "count_vectorizer.fit(X_train)\n",
    "\n",
    "# Transform to counter\n",
    "X_train_counts = count_vectorizer.transform(X_train)\n",
    "X_test_counts = count_vectorizer.transform(X_test)\n",
    "\n",
    "# Model\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train_counts, Y_train)\n",
    "\n",
    "print \"Area under the ROC curve on the test data = %.3f\" % metrics.roc_auc_score(model.predict(X_test_counts), Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also take a look at the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = metrics.roc_curve(Y_test, model.predict_proba(X_test_counts)[:,1])\n",
    "tprs.append(tpr)\n",
    "fprs.append(fpr)\n",
    "roc_labels.append(\"Default Counts\")\n",
    "ax = plt.subplot()\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlabel(\"fpr\")\n",
    "plt.ylabel(\"tpr\")\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlim([0, 1])\n",
    "plt.ylim([0, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tf-idf\n",
    "Another popular technique when dealing with text is to use the term frequency - inverse document frequency (tf-idf) measure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fit a counter\n",
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "tfidf_vectorizer.fit(X_train)\n",
    "\n",
    "# Transform to a counter\n",
    "X_train_tfidf = tfidf_vectorizer.transform(X_train)\n",
    "X_test_tfidf = tfidf_vectorizer.transform(X_test)\n",
    "\n",
    "# Model\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train_tfidf, Y_train)\n",
    "\n",
    "print \"Area under the ROC curve on the test data = %.3f\" % metrics.roc_auc_score(model.predict(X_test_counts), Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we can look at the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fpr, tpr, thresholds = metrics.roc_curve(Y_test, model.predict_proba(X_test_tfidf)[:,1])\n",
    "tprs.append(tpr)\n",
    "fprs.append(fpr)\n",
    "roc_labels.append(\"Default Tfidf\")\n",
    "ax = plt.subplot()\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlabel(\"fpr\")\n",
    "plt.ylabel(\"tpr\")\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlim([0, 1])\n",
    "plt.ylim([0, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `CountVectorizer()` and `TfidfVectorizer()` functions have many options. You can restrict the words you would like in the vocabulary. You can add n-grams. You can use stop word lists. Which options you should use generally depend on the type of data you are dealing with. We can discuss and try some of them now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a few different feature sets and models, let's look at all of our ROC curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for fpr, tpr, roc_label in zip(fprs, tprs, roc_labels):\n",
    "    plt.plot(fpr, tpr, label=roc_label)\n",
    "\n",
    "plt.xlabel(\"fpr\")\n",
    "plt.ylabel(\"tpr\")\n",
    "plt.title(\"ROC Curves\")\n",
    "plt.legend()\n",
    "plt.xlim([0, .07])\n",
    "plt.ylim([.98, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### *A note on cross validation*\n",
    "We didn't use cross validation here, but it is definitely possible. The code is a little messier, so we will leave this to a Forum discussion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes\n",
    "So far we have been exposed to tree classifiers and logistic regression in class. We have also seen SVMs in the homwork. A popular modeling technique (especially in text classification) is the is the (Bernoulli) naive Bayes classifier.\n",
    "\n",
    "Using this model in sklearn is just as easy as all the other models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = BernoulliNB()\n",
    "model.fit(X_train_tfidf, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"AUC on the count data = %.3f\" % metrics.roc_auc_score(model.predict(X_test_tfidf), Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The past few weeks we have seen that many of the models we are using have different parameters that can be tweaked. In naive Bayes, the parameter that is typically tuned is the Laplace smoothing value `alpha`. We won't discuss this in class, but will post a discussion on the NYU Classes Forum.  Also, there is another version of naive Bayes (not discussed in the book) called multinomial naive Bayes, which can handle count features and not just binary features.  We will give an additional reading covering that (in the Forum as well).  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
